Input/output device

ABSTRACT

An input/output device capable of displaying a high-resolution two-dimensional image and detecting the position of an object in three-dimensional space in spite of having a low profile is provided. An input/output device includes: a display section including, in a mixed manner, image pickup regions for obtaining image pickup data of an object and a display region for displaying an image based on image data; a microlens arranged on a display surface of the display section so as to correspond to each of the image pickup regions; and a position detection section detecting position of the object based on image pickup data obtained from the image pickup regions of the display section.

CROSS REFERENCES TO RELATED APPLICATIONS

This is a Continuation application of U.S. patent application Ser. No.12/461,173, filed Aug. 4, 2009, which in turn claims priority fromJapanese Patent Application No.: 2008-221815 filed in the Japan PatentOffice on Aug. 29, 2008, the entire contents of which being incorporatedherein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an input/output device used for, forexample, a touch panel or the like, and capable of inputting andoutputting an image.

2. Description of the Related Art

In related art, an input/output device capable of inputting andoutputting an image in three-dimensional space has been proposed asdescribed in Japanese Unexamined Patent Application Publication No.2007-86931. In Japanese Unexamined Patent Application Publication No.2007-86931, a microlens array is arranged on a display surface side of adisplay at a distance equal to the focal length of the microlens arrayfrom the display surface, thereby three-dimensional display by aso-called integral system is achieved. In such a technique, in aplurality of pixels forming the display, the number of pixels allocatedto one microlens corresponds to the resolution in a depth direction of athree-dimensional image, and the number of microlenses corresponds tothe number of pixels on a two-dimensional plane of the three-dimensionalimage.

SUMMARY OF THE INVENTION

In other words, in a technique in Japanese Unexamined Patent ApplicationPublication No. 2007-86931, the resolution on a two-dimensional plane ofa three-dimensional image is determined depending on the number ofmicrolenses (a lens pitch) in a microlens array. As a result, theresolution is low due to restrictions by the lens pitch.

Therefore, an input/output device of displaying a pseudo-stereoscopicimage by arranging a microlens array on a display surface side of adisplay so as to have an optically conjugate positional relationshipbetween the display surface of the display and one plane in space,thereby to display a two-dimensional image on the one plane in space hasbeen proposed as described in Japanese Unexamined Patent ApplicationPublication No. 2005-141102. In such a configuration, the resolution ofthe image displayed on the one plane in space is adjusted by a pitchbetween display pixels in the display and the imaging magnification of amicrolens, so a high-resolution image is easily displayed.

However, in the configuration in Japanese Unexamined Patent ApplicationPublication No. 2005-141102, a position detection sensor detecting theposition of a finger or the like is arranged forward of an image displaysurface (on a viewer's side), so there is an issue that the thickness ofthe whole device is increased.

It is desirable to provide an input/output device displaying ahigh-resolution two-dimensional image and being capable of detecting theposition of an object in three-dimensional space in spite of having alow profile.

According to an embodiment of the invention, there is provided aninput/output device including: a display section including, in a mixedmanner, image pickup regions for obtaining image pickup data of anobject and a display region for displaying an image based on image data;a microlens arranged on a display surface of the display section so asto correspond to each of the image pickup regions; and a positiondetection section detecting position of the object based on image pickupdata obtained from the image pickup regions of the display section.

In the input/output device according to the embodiment of the invention,when an image is displayed on the display region of the display section,image light is emitted to above the display section. At this time, whenthe object approaches above the display section, the above-describedimage light is applied to the object, and the applied light is condensedby the microlens, and then is received in the image pickup regions.Thereby, in the image pickup regions, image pickup data of the objectkeeping information on the traveling direction of a light ray isobtained. In the position detection section, the position of the objectis detected based on the image pickup data obtained in such a manner.

In the input/output device according to the embodiment of the invention,the display region and image pickup regions are arranged in a mixedmanner in the display section, so compared to the case where a positionsensor for detecting the position of an object is arranged forward of adisplay surface, the thickness of the whole device is reduced. Moreover,the microlens is arranged corresponding to each of the image pickupregions, so compared to the case where a three-dimensional image isdisplayed by a microlens array including microlenses on the wholesurface of the display section, a high-resolution two-dimensional imageis displayed. On the other hand, image pickup data keeping informationon the traveling direction of a light ray is obtainable, so in theposition detection section, not only a two-dimensional plane of anobject but also the position in a height direction of the object isdetectable based on such image pickup data. Therefore, the input/outputdevice is allowed to display a high-resolution two-dimensional image anddetect the position of an object in three-dimensional space in spite ofhaving a low profile.

Other and further objects, features and advantages of the invention willappear more fully from the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of the whole configuration of aninput/output device according to an embodiment of the invention.

FIG. 2 is a schematic sectional view of a brief configuration of aninput/output panel illustrated in FIG. 1.

FIG. 3 is a schematic view of a two-dimensional configuration of adisplay section in FIG. 1.

FIGS. 4A to 4C are schematic views for describing parallax imageproducing operation of an image processing section illustrated in FIG.1.

FIG. 5A is an actually picked-up image, and FIGS. 5B and 5C are parallaximages.

FIGS. 6A to 6C are schematic views for describing position detectingoperation of a position detection section illustrated in FIG. 1.

FIG. 7 is parallax images of a stylus.

FIG. 8 is a perspective view of a brief configuration of a cellularphone according to an application example of the input/output deviceillustrated in FIG. 1.

FIG. 9 is a perspective view of a brief configuration of a cellularphone according to a modification.

FIGS. 10A and 10B are schematic sectional views of another example ofthe input/output panel illustrated in FIG. 1.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A preferred embodiment will be described in detail below referring tothe accompanying drawings.

FIG. 1 illustrates a functional block diagram of an input/output device1 according to an embodiment of the invention. The input/output device 1is used as, for example, a touch panel, and displays an image such as agraphic form or a character, and picks up an image of an object such as,for example, a finger to allow detection (sensing) of the position ofthe object. The input/output device 1 includes an input/output panel 10,a display signal driver 21, a display-side scanner 22, an image pickupsignal receiver 31, an image pickup-side scanner 32, an image processingsection 40, a position detection section 50 and a control section 60.

The input/output panel 10 includes a lens substrate 12 which will bedescribed later on a display side, that is, a viewer's side of a displaysection 11 in which a plurality of pixels are arranged in a matrix formon the whole surface thereof. The configuration of the input/outputpanel 10 will be described in detail later.

The display signal driver 21 and the display-side scanner 22 drive eachdisplay pixel in the display section 11 which will be described later inresponse to the control of the control section 60 to perform displaybased on a display signal (image data D0). The display signal driver 21drives each display pixel subjected to display drive, and thedisplay-side scanner 22 selects the display pixel subjected to displaydrive.

The image pickup signal receiver 31 and the image pickup-side scanner 32drive each image pickup pixel in the display section 11 which will bedescribed later in response to the control of the control section 60 toreceive light from the object thereon, and then obtain image pickup dataD1 of the object. The image pickup signal receiver 31 drives imagepickup pixels subjected to image pickup drive, and the image pickup-sidescanner 32 selects the image pickup pixels subjected to image pickupdrive.

The image processing section 40 performs predetermined image processingon the image pickup data D1 supplied from the image pickup signalreceiver 31 in response to the control of the control section 60 toproduce image processing data D2. The image processing operation of theimage processing section 40 will be described in detail later.

The position detection section 50 detects the position (x, y, z) of afinger based on the image processing data D2 inputted from the imageprocessing section 40 in response to the control of the control section60. Information on the position of the finger detected by the positiondetection section 50 is outputted as position data Dout.

The control section 60 controls the operations of the display signaldriver 21, the display-side scanner 22, the image pickup-side scanner32, the image pickup signal receiver 31, the image processing section 40and the position detection section 50. More specifically, the controlsection 60 appropriately controls display drive operation of the displaysignal driver 21 and the display-side scanner 22, image pickup driveoperation of the image pickup signal receiver 31 and the imagepickup-side scanner 32, image processing operation of the imageprocessing section 40 and position detecting operation of the positiondetection section 60. In addition, the above-described image processingsection 40, the above-described position detection section 50 and theabove-described control section 60 each include, for example, amicrocomputer or the like.

Next, referring to FIGS. 2 and 3, the configuration of the input/outputpanel 10 will be described in detail below. FIG. 2 illustrates aschematic sectional view of the input/output panel 10, and FIG. 3illustrates a schematic plan view of the display section 11.

In the display section 11, display regions 11A including display pixelsPa and image pickup regions 11B including image pickup pixels Pb arearranged on a drive substrate (not illustrated) in a mixed manner. Thedrive substrate drives both of the display pixels Pa and the imagepickup pixels Pb, and is made of, for example, a TFT (Thin FilmTransistor) or the like.

The display pixels Pa are display elements displaying an image based onthe image data D0. The display pixels Pa are made of, for example,liquid crystal display elements (LCDs: liquid crystal displays). Theliquid crystal display elements each includes, for example, a backlight,a liquid crystal cell including a liquid crystal layer sealed between apair of substrates, a pair of polarizing plates bonded to the liquidcrystal cell on a light incident side and a light emission side, and acolor filter selectively allowing light in a specific wavelength regionof light emitted from the liquid crystal cell to pass therethrough (allnot illustrated). The backlight is made of, for example, a CCFL (a ColdCathode Fluorescent Lamp), an LED (Light Emitting Diode) or the like. Insuch a configuration, when a voltage is applied to the pair ofsubstrates in the liquid crystal cell, light emitted from the backlightbased on the image data D0 is modulated, and is allowed to pass thoughthe liquid crystal cell.

The image pickup pixels Pb obtain image pickup data D1 based on lightcondensed by microlens sections 12 b of the lens substrate 12. The imagepickup pixels Pb are made of, for example, CCDs (Charge CoupledDevices), CMOSs (Complementary Metal Oxide Semiconductors) or the like.

The lens substrate 12 includes the microlens sections 12 b in aplurality of positions separated from one another on a substrate made ofglass or the like. In the lens substrate 12, parts other than themicrolens sections 12 b are flat sections 12 a.

Each display region 11A of the display section 11 is arranged opposed toeach flat section 12 a of the lens substrate 12, and each image pickupregion 11B is arranged opposed to each microlens section 12 b. Forexample, as illustrated in FIG. 3, a plurality of image pickup regions11B are arranged at predetermined intervals, and each image pickupregion 11B includes 2×2=4 image pickup pixels Pb. In other words, themicrolens sections 12 b are arranged only opposed to the image pickupregions 11B.

Next, functions and effects of the above-described input/output device 1will be described below.

Image Display Operation

First, image display operation (output operation) of the input/outputdevice 1 will be described below. In the input/output device 1, when theimage data D0 is supplied to the display signal driver 21 in response tothe control of the control section 60, each of the display pixels Pa isdriven to display an image in the display region 11A. Image light of thedisplayed image passes through the flat sections 12 a of the lenssubstrate 12 to be emitted to above the lens substrate 12.

Sensing Operation

Next, sensing operation (input operation) of the input/output device 1will be described referring to FIGS. 1 to FIGS. 6A to 6C. FIGS. 4A to 4Care schematic views for describing parallax image producing operation inthe image processing section 40. FIGS. 5A to 5C are an actuallypicked-up image of a finger, and parallax images produced from thepicked-up image. FIGS. 6A to 6C are illustrations for describingposition detecting operation.

In the input/output device 1, when a finger 100 is placed above theinput/output panel 10, the finger 100 is irradiated with theabove-described image light. The light applied to the finger 100 iscondensed by the microlens sections 12 b in the lens substrate 12, andthen is received in the image pickup regions 11B (the image pickuppixels Pb). Thereby, image pickup data of the finger 100 is obtained,and is outputted to the image pickup signal receiver 31. The imagepickup data outputted to the image pickup signal receiver 31 is suppliedto the image processing section 40 as the image pickup data D1.

In this case, in each image pickup region 11B, the traveling directionsof light rays received by the image pickup pixels Pb are different fromone another, and the light rays keeping information on the travelingdirections are received. Moreover, image pickup pixel data in the imagepickup pixels Pb located at the same position in the image pickupregions 11B include information on the same traveling direction.

The image processing section 40 performs a parallax image producingprocess based on the inputted image pickup data D1 to produce twoparallax image data D11 and D12 of the finger 100. More specifically,first, image pickup pixel data in the image pickup pixels Pb1 located atthe same position in the image pickup regions 11B (for example, pixeldata at positions indicated by a reference numeral 1 in FIG. 4A) areextracted from the image pickup data D1, and the extracted image pickuppixel data are synthesized to produce parallax image data D11illustrated in FIG. 4B. Likewise, image pickup pixel data in the imagepickup pixels Pb2 (pixel data at positions indicated by a referencenumeral 2 in FIG. 4A) are extracted, and the extracted image pickuppixel data are synthesized to produce parallax image data D12illustrated in FIG. 4C.

The produced parallax image data D11 and D12 are subjected to anotherimage processing, for example, a defect correction process or a noisereduction process as necessary, and then the parallax image data D11 andD12 are outputted to the position detection section 50 as imageprocessing data D2.

The position detection section 50 detects the position (x, y, z) of thefinger 100 based on the inputted image processing data D2. For example,the position (x, y) of the finger 100 is detected by performing an edgedetection process on one parallax image data (the parallax image dataD11 or the parallax image data D12) of the finger 100. On the otherhand, the z component (height H) of the position of the finger 100 isspecified by, for example, the following technique. A phase differencebetween living body images in two parallax images is calculated based ona correlation between the parallax image data D11 and D12, and the zcomponent of the finger 100 is specified based on the phase difference.FIGS. 6A to 6C illustrate parallax images corresponding to the parallaximage data D11 and D12 in the case where the height H is 0 mm, 5 mm and10 mm. As illustrated in FIGS. 6A to 6C, phase differences (Δδ₁, Δδ₂ andΔδ₃) between the phases of the finger 100 in the two parallax images aredetected by, for example, the edge detection process. At this time, itis obvious that as the height H of the finger 100 is reduced, the phasedifference between the phases of the finger 100 in the two parallaximages is reduced. (Δδ₃>Δδ₂>Δδ₁). Therefore, when the phase differencebetween the phases of the finger 100 in two parallax images is detectedbased on the correlation between the two parallax images, the zcomponent of the finger 100 is specified.

Moreover, for example, a plurality of parallax image data of the finger100 are successively produced, and the edge detection process isperformed on each of the plurality of parallax image data to calculatethe movement amount of the finger 100, thereby the movement of thefinger 100 in a horizontal plane (in an XY plane) may be detected.

Information on the position (x, y, z) of the finger 100 detected in theposition detection section 50 in such a manner is outputted to outsideas position data Dout. On the other hand, in the input/output device 1,when function operation which is executed in response to the position ofthe finger 100 is set in advance, desired function operation is executedin response to the outputted position data Dout. Alternatively, theposition data Dout may be inputted to the display signal driver 21 asthe above-described image data D0.

As described above, in the embodiment, the display regions 11A and theimage pickup regions 11B are arranged in the display section 11 in amixed manner, so compared to the case where a position sensor fordetecting the position of the finger 100 is separately arranged forwardof the display surface, the thickness of the whole device is reduced.Moreover, the flat sections 12 a of the lens substrate 12 are arrangedopposed to the display regions 11A, so compared to the case where athree-dimensional image is displayed through the microlens array, ahigh-resolution two-dimensional image is displayed. On the other hand,the microlens sections 12 b of the lens substrate 12 are arrangedopposed to the image pickup regions 11B, so image pickup data keepinginformation on the traveling direction of a light ray is obtained.Therefore, the position detection section 50 is allowed to detect notonly a two-dimensional plane of the finger 100 but also the position ina height direction of the finger 100. Therefore, in spite of having alow profile, the input/output device 1 displays a high-resolutiontwo-dimensional image, and detects the position of an object inthree-dimensional space.

In addition, in the above-described embodiment, the case where theposition of the finger 100 is detected is described as an example.However, an object subjected to position detection is not limited to thefinger, and may be any other object such as other parts of a living bodyor a stylus. FIG. 7 illustrates two parallax images produced based onimage pickup data of the stylus. Also in this case, the position (z, y,z) of the stylus is detectable by detecting a phase difference Δδsbetween the phases of the stylus in the two parallax images. Therefore,three-dimensional sensing using not only the finger but also the stylusis allowed.

Application Example 1

FIG. 8 illustrates a brief configuration of a cellular phone 2 accordingto Application Example 1 of the above-described embodiment. The cellularphone 2 includes a flip-type enclosure (a first enclosure 23 and asecond enclosure 24), and the input/output device 1 is arranged in thefirst enclosure 23. On the other hand, in the second enclosure 24, anoperation section 25 for performing input operation is arranged.

In the application example, the input/output device 1 is mounted,thereby the input/output device 1 is allowed to function as a touchpanel from which the input operation is performed in response to theposition of the finger 100 of a user. More specifically, when the zcomponent of the position of the finger 100 is detected, input by clickoperation is allowed. Moreover, when the frequency of occurrences ofcontact by the finger 100 in a certain period is measured, input bydouble-click operation is allowed. Further, when the movement of thefinger 100, that is, a change in the position (x, y) is detected, theinput/output device 1 may function as a mouse pointer.

In addition, an apparatus to which the input/output device 1 is appliedis not limited to the above-described cellular phone, and theinput/output device 1 is applicable to, for example, various mobiledevices such as a notebook type PC (Personal Computer).

Application Example 2

FIG. 9 illustrates a brief configuration of a cellular phone 3 accordingto Application Example 2 of the above-described embodiment. The cellularphone 3 in the application example has the same configuration of thecellular phone 2 according to the above-described Application Example 1,except that light sources 26 are arranged. The light sources 26 arearranged at both sides of the input/output device 1 in the firstenclosure 23, and are made of, for example, LEDs or the like. In theabove-described embodiment and Application Example 1, the case where thefinger 100 is irradiated with displayed image light is described as anexample, but as in the case of the application example, the lightsources 26 for sensing may be separately arranged. Thereby,three-dimensional sensing of the finger 100 is allowed, for example,even under circumstances where an image is not displayed.

Although the present invention is described referring to the embodimentsand the modifications, the invention is not limited thereto, and may bevariously modified. For example, in the above-described embodiment andthe like, in the image processing section 40, two parallax image dataare produced based on the image pickup data D1. However, the number ofproduced parallax image data is not limited to two, and may be three ormore. Moreover, in the above-described embodiment and the like, the casewhere each image pickup region 11B includes 2×2=4 image pickup pixels Pbis described as an example. However, as the number of image pickuppixels allocated to each image pickup region 11B, two is enough, andthree, or five or more image pickup pixels may be allocated to eachimage pickup region 11B.

Further, in the above-described embodiment and the like, a configurationusing the lens substrate 12, that is, a configuration in which themicrolens sections 12 b are arranged on the substrate together with theflat sections 12 a is described as an example. However, the microlenssections 12 b are not necessarily arranged on the substrate. Forexample, an input/output panel 30 illustrated in FIG. 10A may be used.In the input/output panel 30, the display regions 11A and the imagepickup regions 11B are included in a panel plane in a mixed manner, andmicrolenses 30 a (microlens sections) are integrally arranged with theimage pickup regions 11B only on the image pickup regions 11B. In thiscase, FIG. 10B illustrates an enlarged view around one image pickupregion 11B. The image pickup region 11B includes, for example, 2×2 imagepickup pixels Pb, and an on-chip lens 33 for efficiently condensing areceived light ray is arranged on a light-sensing plane of each imagepickup pixel Pb. The microlens 30 a is formed on the on-chip lenses 33with, for example, a resin layer 34 in between. As described above, thelens substrate including the microlenses is not necessarily used, andthe microlens 30 a may be formed directly on the image pickup region 11Bof the panel. Even in such a configuration, the same effects as those ofthe embodiment of the invention are obtainable.

In the above-described embodiment and the like, the liquid crystaldisplay element is described as an example of the display pixel Pa.However, any other display element, for example, a self-luminous elementsuch as an organic or inorganic EL (Electro Luminescence) element may beused. However, in the case where the self-luminous element is used, thebacklight may not be specifically arranged.

The present application contains subject matter related to thatdisclosed in Japanese Priority Patent Application JP 2008-221815 filedin the Japan Patent Office on Aug. 29, 2008, the entire content of whichis hereby incorporated by reference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

1. An input/output device comprising: a display section including, in amixed manner, image pickup regions configured to obtain image pickupdata of an object and a display region configured to display an imagebased on image data; a microlens arranged on a display surface of thedisplay section selectively opposed to each of the image pickup regions;and a position detection section detecting position of the object basedon image pickup data obtained from the image pickup regions of thedisplay section; wherein in the display section, the image pickupregions are arranged in a plurality of positions separated from oneanother, and each of the image pickup regions includes two or more imagepickup pixels.
 2. The input/output device according to claim 1, whereinthe microlens is formed on a substrate, and the substrate has a flatportion in a region corresponding to the display region.
 3. Theinput/output device according to claim 1, wherein the microlens isintegrally arranged with the image pickup regions on the image pickupregions of the display section.
 4. The input/output device according toclaim 4, further comprising: an image processing section performingimage processing on image pickup data obtained from the image pickupregions, wherein the image processing section extracting image pickuppixel data from the image pickup data, each of the image pickup pixeldata corresponding to each of the image pickup pixel located at the sameposition in the image pickup regions, and then synthesizing the imagepickup pixel data to produce two or more parallax image data.
 5. Theinput/output device according to claim 4, wherein the position detectionsection detects the position of the object based on a phase differencebetween a couple of parallax image data.